Repository navigation
[schema-descriptions] Carry Choice and Score question text inside the schema definition - #31
Merged
Merged
Conversation
Choice and Score answers are nested probability-map models, which the JSON
schema emits as a $ref with the question and its option criteria in a
description beside the reference. pydantic-ai's OpenAI transformer inlines
the $ref and merges that description; its Anthropic transformer keeps $defs
and the description beside the $ref does not reach the model. Captured
requests to Anthropic showed intent as a bare $ref whose definition listed
eight option keys described only as {maximum: 1, minimum: 0}: the model
saw no question and no criteria, and answered from the key names.
Set the description as the probability-map model's docstring as well, so
pydantic emits it as the definition's own description, inside $defs, where
it survives either treatment of the reference. Verified on a captured
request to claude-opus-5: the question text is present, and a refund case
the model read as delivery_problem 0.60 with the text missing reads as
refund_request 0.82 with it present, matching a direct API call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Choice and Score answers are nested probability-map models, which the JSON schema emits as a $ref with the question and its option criteria in a description beside the reference. pydantic-ai's OpenAI transformer inlines the $ref and merges that description; its Anthropic transformer keeps $defs and the description beside the $ref does not reach the model. Captured requests to Anthropic showed intent as a bare $ref whose definition listed eight option keys described only as {maximum: 1, minimum: 0}: the model saw no question and no criteria, and answered from the key names.
Set the description as the probability-map model's docstring as well, so pydantic emits it as the definition's own description, inside $defs, where it survives either treatment of the reference. Verified on a captured request to claude-opus-5: the question text is present, and a refund case the model read as delivery_problem 0.60 with the text missing reads as refund_request 0.82 with it present, matching a direct API call.
Before the fix (main). The intent property is a bare reference, and the definition it points to has no description:
The model's answer: refund_request 0.49, delivery_problem 0.49. A coin toss between the two names.
After the fix (fix/choice-score-descriptions-survive-ref). The property is the same bare reference, but the definition now carries the question and every option's criteria: